Information processing apparatus, eye open/closed degree determination method, computer-readable storage medium, and image sensing apparatus

ABSTRACT

An information processing apparatus inputs an image, detects the face of a person from the input image, and calculates a feature amount associated with the open/closed state of eyes of the detected face. In addition, the information processing apparatus calculates, as a feature-change amount, the difference between the calculated feature amount and a predetermined feature amount, and calculates the eye open/closed degree of eyes of the detected face on the basis of the feature amount and the feature-change amount.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an information processing apparatus,eye open/closed degree determination method, computer-readable storagemedium, and image sensing apparatus.

2. Description of the Related Art

A technique of detecting a face from an image including still and movingimages has conventionally been known (Mitarai, Y., Mori, K., Matsugu, M.“Robust Face Detection System Based on Convolutional Neural NetworksUsing Selective Activation of Modules”, FIT (Forum of InformationTechnology), L1-013, 2003). Further, techniques have been proposed fordetermining the open/closed degree of eyes in a detected face.

In this regard, Japanese Patent Laid-Open No. 09-044685 proposes atechnique of comparing the distance between the eyebrow and the pupil orthe area ratio of the eyebrow and pupil with a predetermined reference,and calculating the change amount of the distance, thereby determiningthe eye open/closed degree.

Japanese Patent Laid-Open No. 11-242744 proposes a technique of groupingeye regions into a white region and black (pupil) region, and when theblack region changes, determining that the object blinks.

Japanese Patent Laid-Open No. 2004-192551 proposes a technique ofpreparing a plurality of processes to determine the eye open/closeddegree in order to distinguish an eye state immediately before closingeyes from an eye state upon a change of the expression such as a smile,and determining the eye open/closed degree on the basis of a resultusing the ANDs of respective determination results.

There is an individual difference in eye size (e.g., the distancebetween upper and lower eyelids or the appearance of the pupil) in anexpressionless state while the eyes are fully open. For example, thepupil of person A shown in FIG. 19 is completely seen in anexpressionless state (state 1) while the eyes are fully opened. Incontrast, the pupil of person B shown in FIG. 19 is partially hidden inan expressionless state (state 4) while the eyes are fully opened.

In this manner, there is an individual difference in eye size in anexpressionless state. If the eye open/closed degree is determined fromonly one image, state 2 of person A shown in FIG. 19 and state 4 ofperson B shown in FIG. 19 are determined to have the same eyeopen/closed degree. For example, when state 2 of person A shown in FIG.19 is determined as a shooting failure upon shooting by an image sensingapparatus, state 4 of person B shown in FIG. 19 is also determined as ashooting failure.

When determining the eye open/closed degree using the amount of changebetween a plurality of images, the amount of change from state 1 tostate 2 of person A shown in FIG. 19 equals that of change from state 4to state 5 of person B shown in FIG. 19. It is determined that state 2of person A and state 5 of person B are the same. For example, whenstate 2 of person A shown in FIG. 19 is determined as a shooting successupon shooting by an image sensing apparatus, state 5 of person B shownin FIG. 19 is also determined as a shooting success.

Determination of whether an image shot by an image sensing apparatus isa failure or success changes depending on the user of the image sensingapparatus. It is necessary to detect the eye open/closed degree at highprecision regardless of individual differences in the eye size. Forexample, the above-mentioned technique disclosed in Japanese PatentLaid-Open No. 2004-192551 determines the eye open/closed degree by usinga combination of eye open/closed degree determination results. However,each eye open/closed degree determination result is a binary outputrepresenting that the eyes are open or closed. For this reason, eventhis technique cannot detect the eye open/closed degree at highprecision regardless of individual differences in eye size.

A case where the eye open/closed degree is determined by a method usingone image, and a case where it is determined by a method using theamount of change between a plurality of images will be examined. A casewhere the eye open/closed degree is determined when it changes fromstate 1 to state 2 shown in FIG. 20, and a case where the eyeopen/closed degree is determined when it changes from state 4 to state 5shown in FIG. 20 will be exemplified.

When the eyes change from state 1 (black eye area 10) to state 2 (blackeye area 5) shown in FIG. 20, the change amount of the black eye area is5, and the absolute amount of the black eye area in state 2 is 5. Whenan absolute amount threshold for determining that the eyes are closed isset to 6 (it is determined that the eyes are closed when the absoluteamount is smaller than 6), and a change amount threshold for determiningthat the eyes are closed is set to 4 (it is determined that the eyes areclosed when the change amount is larger than 4), both determinationresults based on the absolute amount and change amount represent thatthe eyes are closed. When the eyes change from state 4 (black eye area5) to state 5 (black eye area 0) shown in FIG. 20 and the samedeterminations as those described above are made, the same results areobtained, and both determination results based on the absolute amountand change amount represent that the eyes are closed. In this manner,according to the conventionally proposed eye open/closed degreedetermination method, state 2 and state 5 shown in FIG. 20 aredetermined to have the same eye open/closed degree.

SUMMARY OF THE INVENTION

The present invention enables to provide an information processingapparatus, eye open/closed degree determination method,computer-readable storage medium, and image sensing apparatus capable ofdetecting the eye open/closed degree at high precision regardless ofindividual differences in eye size.

According to a first aspect of the present invention there is providedan information processing apparatus comprising: an input unit configuredto input an image; a face detection unit configured to detect a face ofa person from the image input from the input unit; a first calculationunit configured to calculate a feature amount associated with an eyeopen/closed state from the face detected by the face detection unit; asecond calculation unit configured to calculate, as a feature-changeamount, a difference between the feature amount calculated by the firstcalculation unit and a feature amount calculated from a face detectedfrom an image input before a predetermined time; and a third calculationunit configured to calculate an eye open/closed degree of eyes of theface detected by the face detection unit on the basis of the featureamount and the feature-change amount.

According to a second aspect of the present invention there is providedan eye open/closed degree determination method comprising: inputting animage;

detecting a face of a person from the image input in the inputting theimage; calculating a feature amount associated with an eye open/closedstate from the face detected in the detecting the face; calculating, asa feature-change amount, a difference between the feature amountcalculated in the calculating the feature amount and a feature amountcalculated from a face detected from an image input before apredetermined time; and calculating an eye opening/closing degree ofeyes of the face detected in the detecting the face on the basis of thefeature amount and the feature-change amount.

According to a third aspect of the present invention there is provided acomputer-readable storage medium storing a computer program, the programcausing a computer to function as an input unit configured to input animage, a face detection unit configured to detect a face of a personfrom the image input from the input unit, a first calculation unitconfigured to calculate a feature amount associated with an eyeopen/closed state from the face detected by the face detection unit, asecond calculation unit configured to calculate, as a feature-changeamount, a difference between the feature amount calculated by the firstcalculation unit and a feature amount calculated from a face detectedfrom an image input before a predetermined time, and a third calculationunit configured to calculate an eye open/closed degree of eyes of theface detected by the face detection unit on the basis of the featureamount and the feature-change amount.

According to a fourth aspect of the present invention there is providedan image sensing apparatus comprising: an input unit configured to inputan image; a face detection unit configured to detect a face of a personfrom the image input from the input unit; a first calculation unitconfigured to calculate a feature amount associated with an eyeopen/closed state from the face detected by the face detection unit; asecond calculation unit configured to calculate, as a feature-changeamount, a difference between the feature amount calculated by the firstcalculation unit and a feature amount calculated from a face detectedfrom an image input before a predetermined time; a third calculationunit configured to calculate an eye open/closed degree of eyes of theface detected by the face detection unit on the basis of the featureamount and the feature-change amount; a determination unit configured toperform eye open/closed determination by performing threshold processingfor the eye open/closed degree calculated by the third calculation unit;and an image sensing unit configured to shoot on the basis of a resultof eye open/closed degree determination by the determination unit.

Further features of the present invention will be apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing an example of the arrangement of animage sensing apparatus 100 according to an embodiment of the presentinvention;

FIGS. 2A and 2B are a flowchart showing an example of a processingsequence in the image sensing apparatus 100 shown in FIG. 1;

FIG. 3 is a flowchart showing an example of the sequence of eyeopen/closed degree determination in step S113 shown in FIG. 2B;

FIG. 4 is a view showing an example of an outline of imagenormalization;

FIG. 5 is a view showing an example of an outline of processing whensetting a region to undergo detection of a pupil feature;

FIG. 6 is a view showing an example of a pupil feature detection result;

FIG. 7 is a graph showing an example of the distribution of eyeopen/closed degree values calculated by a predetermined function func₁;

FIG. 8 is a graph showing an example of the distribution of eyeopen/closed degree values calculated by a predetermined function func₂;

FIG. 9 is a first view for explaining an example of the effects of thefirst embodiment;

FIG. 10 is a second view for explaining an example of the effects of thefirst embodiment;

FIG. 11 is a flowchart showing an example of the sequence of eyeopen/closed degree determination according to the second embodiment;

FIG. 12 is a graph showing an example of a change of the pupil-featureamount within a predetermined time;

FIG. 13 is a first view for explaining an example of the effects of thesecond embodiment;

FIGS. 14A and 14B are a flowchart showing an example of a processingsequence in an image sensing apparatus 100 according to the thirdembodiment;

FIG. 15 is a flowchart showing an example of the sequence of eyeopen/closed degree determination in step S416 shown in FIG. 14B;

FIG. 16 is a flowchart showing an example of the sequence of weightdetermination in step S502 shown in FIG. 15;

FIG. 17 is a first view for explaining an example of the effects of thethird embodiment;

FIG. 18 is a second view for explaining an example of the effects of thethird embodiment;

FIG. 19 is a first view for explaining a conventional problem; and

FIG. 20 is a second view for explaining the conventional problem.

DESCRIPTION OF THE EMBODIMENTS

Preferred embodiments of the present invention will now be described indetail with reference to the drawings. It should be noted that therelative arrangement of the components, the numerical expressions andnumerical values set forth in these embodiments do not limit the scopeof the present invention unless it is specifically stated otherwise.

First Embodiment

FIG. 1 is a block diagram showing an example of the arrangement of animage sensing apparatus 100 according to an embodiment of the presentinvention. The first embodiment will exemplify a case where the imagesensing apparatus 100 is an electronic still camera.

The image sensing apparatus 100 includes an image sensing lens unit 101,and a light amount adjustment device 102 having a stop and shutter. Animage sensor 103 is, for example, a CCD (Charge-Coupled Device) or CMOS(Complementary Metal Oxide Semiconductor) sensor for converting a lightbeam of an object image having passed through the image sensing lensunit into an electrical signal. An analog signal processing circuit 104performs clamp, gain, and the like for an analog signal output from theimage sensor 103. An A/D converter 105 converts an output from theanalog signal processing circuit 104 into a digital signal.

A digital signal processing circuit 107 performs predetermined pixelinterpolation, color conversion, and the like for data from the A/Dconverter 105 or data from a memory control circuit 106. The digitalsignal processing circuit 107 also performs predetermined arithmeticusing sensed image data, and TTL AWB (Auto White Balance) processingbased on the obtained arithmetic result. Further, the digital signalprocessing circuit 107 executes specific object face detection and eyeopen/closed degree determination for the sensed image data. The specificobject face detection and eye open/closed degree determination use datastored in a memory (a memory 108 to be described later).

A system control circuit 112 performs TTL (Through The Lens) AF (AutoFocus), AE (Auto Exposure), and EF (pre-Electronic Flash) on the basisof the above-mentioned arithmetic results to control an exposure controlcircuit 113 and focus control circuit 114.

The memory control circuit 106 controls the analog signal processingcircuit 104, the A/D converter 105, the digital signal processingcircuit 107, the memory 108, and a digital/analog (to be referred to asD/A hereinafter) converter 109. Data A/D-converted by the A/D converter105 is written in the memory 108 via the digital signal processingcircuit 107 and memory control circuit 106. Data A/D-converted by theA/D converter 105 is sometimes written in the memory 108 directly viathe memory control circuit 106.

The memory 108 stores, for example, data to be displayed on a displayapparatus 110. Data stored in the memory 108 is output to the displayapparatus 110 such as a TFT or LCD via the D/A converter 109, anddisplayed on it. The memory 108 stores images including sensed stillimages and moving images. The memory 108 has a capacity enough to storea predetermined number of still images and a moving image of apredetermined time. Even in continuous shooting or panoramic shooting ofsuccessively shooting a plurality of still images, many images can bewritten in the memory 108 at high speed. The memory 108 is alsoavailable as the work area of the system control circuit 112. Sensedstill images and moving images may also be written in a storage mediumsuch as a CD-ROM, floppy disk®, hard disk, magnetic tape,magneto-optical disk, or nonvolatile memory card by using an interface(I/F) 111.

The display apparatus 110 can sequentially display sensed image data,and in this case, functions as an electronic viewfinder. The displayapparatus 110 can arbitrarily turn on/off the display in accordance withan instruction from the system control circuit 112. When the display isturned off, power consumption of the image sensing apparatus 100 can bereduced more greatly than when the display is ON. The display apparatus110 displays an operating state, message, and the like using a text,image, or the like in accordance with execution of a program in thesystem control circuit 112.

The I/F 111 interfaces the image sensing apparatus 100 with a storagemedium such as a memory card or hard disk. By using the I/F 111, theimage sensing apparatus 100 can exchange image data and managementinformation accessory to the image data with another computer or aperipheral device such as a printer. When the I/F 111 complies with thestandard of a PCMCIA card, CF (Compact Flash®) card, or the like, theI/F 111 functions as a communication interface upon connecting variouscommunication cards. These communication cards include a LAN card, modemcard, USB card, IEEE1394 card, P1284 card, SCSI card, and PHScommunication card.

The system control circuit 112 controls the overall operation of theimage sensing apparatus 100. The memory in the system control circuit112 stores constants, variables, programs, and the like for operatingthe system control circuit 112, or recognizing the face of a specificobject and the eye open/closed degree. The Constants, variables,programs, and the like stored in the internal memory of the systemcontrol circuit 112 can be changed using a CD-ROM, floppy disk®, harddisk, magnetic tape, magneto-optical disk, nonvolatile memory card, orthe like.

The exposure control circuit 113 controls the stop and shutter of thelight amount adjustment device 102. The focus control circuit 114controls focusing and zooming of the image sensing lens unit 101. Theexposure control circuit 113 and focus control circuit 114 arecontrolled by TTL. The system control circuit 112 controls the exposurecontrol circuit 113 and focus control circuit 114 on the basis of theresult of executing calculation for sensed image data by the digitalsignal processing circuit 107.

An example of a processing sequence in the image sensing apparatus 100shown in FIG. 1 will be explained with reference to FIGS. 2A and 2B. Aprogram for executing this processing is stored in, for example, theinternal memory of the system control circuit 112, and executed underthe control of the system control circuit 112.

This processing starts upon power-on or the like. When the processingstarts, the system control circuit 112 initializes various flags,control variables, and the like in the internal memory (step S100).Then, the system control circuit 112 detects the mode setting state ofthe image sensing apparatus 100. An example of the mode is an eyeclosing prevention shooting mode.

If the system control circuit 112 detects a mode other than a shootingmode (another mode in step S101), the image sensing apparatus 100executes processing corresponding to the selected mode (step S102), andthen the process returns to step S101. If the system control circuit 112detects a shooting mode such as the eye closing prevention shooting mode(shooting mode in step S101), it determines whether an error hasoccurred in the remaining battery level or operating state. If an errorhas occurred (NO in step S103), the system control circuit 112 issues apredetermined warning by an image or sound using the display apparatus110 (step S104), and then the process returns to step S101. If no errorhas occurred (YES in step S103), the system control circuit 112determines whether the storage medium operating state is improper forthe operation of the image sensing apparatus 100, particularly an imagedata recording/playback operation to the storage medium.

If the storage medium operating state is improper (NO in step S105), thesystem control circuit 112 issues a predetermined warning by an image orsound using the display apparatus 110 similarly to the above-mentionedwarning (step S104), and then the process returns to step S101. If thestorage medium is proper (YES in step S105), the system control circuit112 displays a user interface (to be referred to as a UI hereinafter)representing various setting states of the image sensing apparatus 100by an image or sound using the display apparatus 110 (step S106). Whenthe image display of the display apparatus 110 is ON, the UIrepresenting various setting states of the image sensing apparatus 100may also be displayed by an image or sound using the display apparatus110. In this fashion, the user makes various settings.

The system control circuit 112 sets ON the image display of the displayapparatus 110 (step S107). The system control circuit 112 sets a throughdisplay state to sequentially display sensed image data (step S108). Inthe through display state, the display apparatus 110 sequentiallydisplays data written in the memory 108, implementing an electronicviewfinder function.

After that, the image sensing apparatus 100 determines whether the usersuch as a photographer has pressed the shutter switch. If the user hasnot pressed the shutter switch (NO in step S109), the process returns tostep S101. If the user has pressed the shutter switch (YES in stepS109), the system control circuit 112 executes face detection (stepS110). In the face detection, image data input from the image sensor 103is compressed to low resolution, and the face of a person is detectedfrom the compressed image data. More specifically, image data iscompressed to a low resolution of, for example, 640×480 pixels byexecuting thinning or the like, and then a face is detected. A knownexample of the face detection method is convolutional neural network(CNN) of hierarchically detecting high-order features such as an eye andmouth from low-order features such as an edge, and finally detecting thebarycenter of the face (Mitarai, Y., Mori, K., Matsugu, M. “Robust FaceDetection System Based on Convolutional Neural Networks Using SelectiveActivation of Modules”, FIT (Forum of Information Technology), L1-013,2003). By using the CNN, the barycenter of an eye or mouth can beobtained.

After detecting the face of a person in this way, the system controlcircuit 112 executes predetermined AE/AF control for the detectedperson's face (step S111), and through-displays a sensed image (stepS112). At this time, the system control circuit 112 determines the eyeopen/closed degree of each person by using the eye/face positiondetected in step S110 (step S113). Details of the eye open/closed degreedetermination will be explained later.

The system control circuit 112 determines, based on the result of theeye open/closed degree determination, whether to shoot by the imagesensing apparatus 100. More specifically, if the system control circuit112 determines that the eyes are closed, it does not cause the imagesensing apparatus 100 to shoot (No in step S114), and determines whetherto forcibly terminate shooting. If the system control circuit 112determines to forcibly terminate shooting (YES in step S119), theprocess ends. If the system control circuit 112 determines not toforcibly terminate shooting (NO in step S119), it performs facedetection again (step S118), and the process returns to step S112.

If the system control circuit 112 determines that the eyes are open, itcauses the image sensing apparatus 100 to shoot (step S115 after YES instep S114). After shooting, the system control circuit 112 displays aquick review of the sensed image (step S116), encodes the sensedhigh-resolution image, and records the encoded image on a storage mediumsuch as a flash memory (step S117). That is, a low-resolution imagecompressed by thinning or the like is used for face detection, whereas ahigh-resolution image is used for recording.

Details of the eye open/closed degree determination in step S113 shownin FIG. 2B will be explained with reference to FIG. 3.

When this processing starts, the system control circuit 112 executesimage normalization using an eye or face position detected in step S110(step S200). In the normalization, the orientation of each face in aninput image 300 compressed to low resolution is aligned to be erect, asshown in FIG. 4. Then, affine transformation and extraction are done tochange the width between two eyes to a predetermined number of pixels(e.g., 50 pixels) and change the width and height of an extracted imageto a predetermined number of pixels (e.g., 120×120 pixels). In anexample shown in FIG. 4, face images 301 and 302 are obtained as aresult of normalizing the input image 300.

Thereafter, the system control circuit 112 sets an eye region as shownin FIG. 5 in the normalized image (step S201). In an example shown inFIG. 5, the coordinate point of the upper left corner of an eye region303 is set to (1,10), and that of its lower right corner is set to(120,60).

After setting the eye region, the system control circuit 112 performsimage correction (e.g., luminance correction) for the eye region (stepS202). In the luminance correction, the luminance histogram of the eyeregion is created and expanded to change the luminance value of eachpixel.

After the luminance correction, the system control circuit 112 detects apupil feature from the eye region of the luminance-corrected image (stepS203). The pupil feature is detected using a pupil detection CNN whichhas learned in advance to detect the pupil, similar to face detection.The pupil detection CNN is given a pupil region as a right answer and aregion other than the pupil region as a wrong answer, and learns tooutput a large value from each neuron of the CNN in only a region wherethe pupil exists. If the image having undergone luminance correction instep S202 is input to the pupil detection CNN, for example, neuronoutput values as shown in FIG. 6 are obtained.

Then, the system control circuit 112 calculates a pupil-feature amountusing the pupil-feature detection amount (step S204). The pupil-featureamount in the first embodiment is represented by an output valuecounter(t) which becomes equal to or larger than a predeterminedthreshold I_(th) at time t[s]:

if I(x,y)≧I _(th)

Counter(t)+=1   (1)

Instead of the pupil-feature amount, the distance between edgescorresponding to upper and lower eyelids is also available. Apupil-feature amount extracted from one image to undergo determinationof the eye open/closed degree is represented by Feature1(t):

Feature1(t)=Counter(t)   (2)

After the feature amount calculation, the system control circuit 112calculates a feature-change amount Feature2(t) of the calculatedpupil-feature amount Feature1(t) (step S205). The feature-change amountFeature2(t) is calculated by calculating the difference between thepupil-feature amount Feature1(t) and a predetermined feature amount (inthis case, a previously calculated pupil-feature amount Feature1(t_(n)):

Feature2(t)=|Counter(t _(n))−Counter(t)|  (3)

The previously calculated pupil-feature amount Feature1(t_(n)) is apupil-feature amount calculated a predetermined time (e.g., 500 ms)before calculating the pupil-feature amount Feature1(t). In equation(3), n is an average time when the man blinks unconsciously. Forexample, when the average time when the man closes an open eye is 500ms, n=500 ms is set.

After the feature-change amount calculation, the system control circuit112 calculates eye open/closed degrees for the calculated pupil-featureamount Feature1(t) and feature-change amount Feature2(t), respectively(steps S206 and S207). In the first embodiment, the eye open/closeddegree will be explained using an eye closing degree BlinkDegree(t).However, the eye open/closed degree may also be represented by the eyeopening degree.

An eye open/closed degree BlinkDegree1(t) based on the pupil-featureamount Feature1(t) is calculated by substituting the pupil-featureamount Feature1(t) into a predetermined function func₁:

BlinkDegree1(t)=func₁(Feature1(t))   (4)

An eye open/closed degree BlinkDegree2(t) based on the feature-changeamount Feature2(t) is calculated by substituting the feature-changeamount Feature2(t) into a predetermined function func₂:

BlinkDegree2(t)=func₂(Feature2(t))   (5)

As represented by equation (6), the system control circuit 112 addsweights w₁ and w₂ to the eye open/closed degrees BlinkDegree1(t) andBlinkDegree2(t), and then adds the weighted eye open/closed degreesBlinkDegree1(t) and BlinkDegree2(t) (step S208), thereby calculating afinal eye open/closed degree BlinkDegree(t) (step S209):

BlinkDegree(t)=w ₁×BlinkDegree1(t)+w ₂×BlinkDegree2(t)   (6)

where the weights w₁ and W₂ in the embodiment are 1:1.

FIG. 7 shows the distribution of eye open/closed degree valuescalculated by the predetermined function func₁. As shown in FIG. 7, asthe pupil-feature amount Feature1(t) calculated from one image toundergo determination of the eye open/closed degree becomes larger, theeye open/closed degree BlinkDegree1(t) becomes smaller. That is, as thepupil-feature amount Feature1(t) becomes larger, the possibility ofdetermining that the eyes are open becomes higher.

FIG. 8 shows the distribution of eye open/closed degree valuescalculated by the predetermined function func₂. As shown in FIG. 8, asthe feature-change amount Feature2(t) becomes larger, the eyeopen/closed degree BlinkDegree2(t) becomes larger. That is, as thefeature-change amount Feature2(t) becomes larger, the possibility ofdetermining that the eyes are kept closed or being opened or closedbecomes higher.

Referring back to FIG. 3, after executing the final eye open/closeddegree calculation, the system control circuit 112 makes a binarydetermination of whether the eyes are closed or open. This determinationis achieved by executing threshold processing for the eye open/closeddegree BlinkDegree(t) (step S210). The system control circuit 112 sendsback a determination result “eyes open” or “eye closed” based on therelationship (equal to and larger or smaller than the threshold) betweenBlinkDegree(t) and the threshold (step S211). Then, the process ends.

FIGS. 9 and 10 are views for explaining effects obtained using thepupil-feature amount Feature1(t) calculated from an image to undergodetermination of the eye open/closed degree and the correspondingfeature-change amount Feature2(t) when determining the eye open/closedstate.

FIG. 9 is a view showing results of calculating the eye open/closeddegrees in respective states of person A having a large eye open/closedwidth and those of person B having a small eye open/closed width. Asdescribed above, the eye open/closed degree BlinkDegree1(t) iscalculated based on the pupil-feature amount Feature1(t) of an image toundergo determination of the eye open/closed degree. The eye open/closeddegree BlinkDegree2(t) is calculated based on the feature-change amountFeature2(t) representing the difference between the pupil-feature amountFeature1(t) and the previously calculated pupil-feature amount Feature1(t_(n)). BlinkDegree (t) represents the final eye open/closed degreewhich is calculated by adding the weighted BlinkDegree1(t) andBlinkDegree2(t).

A case where the eyes of person A change from state 1 (eye open state)to state 1′ (eye open state) will be examined. In this case, the pupilis clearly seen in state 1′ (eye open state), so the pupil-featureamount Feature1(t) takes a large value. Hence, the eye open/closeddegree BlinkDegree1(t) takes a small value. When the eyes of person Achange from state 1 (eye open state) to state 1′ (eye open state), thepupil-feature amount Feature1(t) hardly changes. Thus, the eyeopen/closed degree BlinkDegree2(t) calculated based on thefeature-change amount Feature2(t) between the pupil-feature amountFeature1(t) and the previously calculated pupil-feature amountFeature1(t_(n)) also takes a small value. As a result, the eyeopen/closed degree BlinkDegree(t) finally calculated after adding theweights=1:1 becomes “0.0”.

A case where the eyes of person A change from state 1 (eye open state)to state 2 (eye half-open state) will be examined. In this case, thepupil region is partially seen in state 2 (eye half-open state), so thepupil-feature amount Feature1(t) takes an intermediate value (e.g., avalue between 0 and 1). Hence, the eye open/closed degreeBlinkDegree1(t) also takes an intermediate value. When the eyes ofperson A change from state 1 (eye open state) to state 2 (eye half-openstate), the pupil-feature amount Feature1(t) slightly changes. Thus, theeye open/closed degree BlinkDegree2(t) calculated based on thefeature-change amount Feature2(t) between the pupil-feature amountFeature1(t) and the previously calculated pupil-feature amountFeature1(t_(n)) also takes an intermediate value. The eye open/closeddegree BlinkDegree(t) finally calculated after adding the weights=1:1becomes “1.0”.

Similarly, a case where the eyes of person A change from state 1 (eyeopen state) to state 3 (eye closed state) will be examined. In thiscase, the eye open/closed degree BlinkDegree1(t) calculated based on thepupil-feature amount Feature1(t) takes a large value. Also, the eyeopen/closed degree BlinkDegree2(t) calculated based on thefeature-change amount Feature2(t) also takes a large value. The eyeopen/closed degree BlinkDegree(t) finally calculated after adding theweights=1:1 becomes “2.0”.

In contrast, a case where the eyes of person B change from state 4 (eyeopen state) to state 4′ (eye open state) will be examined. In this case,the eye open/closed degree BlinkDegree1(t) calculated based on thepupil-feature amount Feature1(t) takes an intermediate value. The eyeopen/closed degree BlinkDegree2(t) calculated based on thefeature-change amount Feature2(t) takes a small value. The eyeopen/closed degree BlinkDegree(t) finally calculated after adding theweights=1:1 becomes “0.5”.

A case where the eyes of person B change from state 4 (eye open state)to state 5 (eye closed state) will be examined. In this case, the eyeopen/closed degree BlinkDegree1(t) calculated based on the pupil-featureamount Feature1(t) takes a large value. The eye open/closed degreeBlinkDegree2(t) calculated based on the feature-change amountFeature2(t) takes an intermediate value. The eye open/closed degreeBlinkDegree(t) finally calculated after adding the weights=1:1 becomes“1.5”.

FIG. 10 shows the distribution of the calculated eye open/closed degreeBlinkDegree(t). As shown in FIG. 10, calculated eye open/closed degreesin the respective states of persons A and B take different values. Inother words, the eye open/closed degree can be detected at highprecision regardless of individual differences in eye size.

As described above, according to the first embodiment, eye open/closeddegrees calculated based on a pupil-feature amount obtained from animage to be determined, and a feature-change amount representing thedifference between the pupil-feature amount and a previously calculatedpupil-feature amount are weighted and added to calculate the final eyeopen/closed degree. As a result, the eye open/closed state can bedetected at high precision regardless of individual differences in eyesize.

In the above-described first embodiment, the final eye open/closeddegree is calculated after weighting. However, weighting is notindispensable, and the final eye open/closed degree may also becalculated without weighting.

Second Embodiment

The second embodiment will now be explained. The arrangement of anapparatus and the sequence of the overall operation in the secondembodiment are the same as those in FIGS. 1 and 2 described in the firstembodiment, a description thereof will be omitted, and only differencesfrom the first embodiment will be explained. The second embodiment isdifferent from the first embodiment in eye open/closed degreedetermination described with reference to FIG. 3.

The sequence of eye open/closed degree determination according to thesecond embodiment will be explained with reference to FIG. 11. In thesecond embodiment, to prevent a repetitive description, a description ofdetailed processing (e.g., methods of detecting and calculating apupil-feature amount) described in the first embodiment will be omitted,and only a rough sequence representing the difference will be explained.

When this processing starts, a system control circuit 112 estimates astate in which the eyes of a face detected in step S110 (FIG. 2A) areopen. More specifically, the system control circuit 112 estimates astate in which the eyes are open, and checks whether this eye statecorresponds to a predetermined eye state (narrow eyes in the secondembodiment). Even when a person with narrow eyes normally opens hiseyes, most part of the pupil region is hidden.

To estimate an eye state when the eyes are open, the system controlcircuit 112 calculates a pupil-feature amount Feature1(t) by the samemethod as that in the first embodiment (step S300). The pupil-featureamount Feature1(t) is repetitively calculated till the lapse of apredetermined time (NO in step S301). That is, the system controlcircuit 112 calculates pupil-feature amounts Feature1(t) from respectiveframe images sequentially input within the predetermined time. Thepredetermined time is an average time taken to blink. It suffices toalways blink within a predetermined time (e.g., 5 sec).

Upon the lapse of the predetermined time (YES in step S301), the systemcontrol circuit 112 estimates an eye state on the basis of thepupil-feature amount Feature1(t) calculated from a plurality of frameimages in step S300 (step S302). More specifically, the system controlcircuit 112 estimates an eye open state by referring to pupil-featureamounts Feature1(t₁), . . . , Feature1(t_(n1)) calculated within thepredetermined time (time t=t₁[s] to time t=t_(n1)[s]).

FIG. 12 is a graph showing the pupil-feature amounts Feature1(t₁), . . .. , Feature1(t_(n1)) calculated from time t=t₁[s] to time t=t_(n1)[s].

Whether the eyes correspond to a predetermined eye state (narrow eyes)is determined based on the width L (=Feature1Max−Feature1Min) of thepupil-feature amount Feature1(t) calculated from t=t₁[s] to timet=t_(n1)[s]. More specifically, if the width L of the pupil-featureamount Feature1(t) is larger than a predetermined threshold L_(th), itis determined that most part of the pupil region is not hidden when theeyes are open. If the width L of the pupil-feature amount Feature1(t) isequal to or smaller than the predetermined threshold L_(th), it isdetermined that most part of the pupil region is hidden when the eyesare open.

If the eyes are not in the predetermined eye state (NO in step S303),the system control circuit 112 calculates BlinkDegree(t_(n1)) using onlythe pupil-feature amount Feature1(t_(n1)) calculated from one image attime t=t_(n1)[s] (step S304). Calculating the eye open/closed degreeBlinkDegree(t_(n1)) using only the pupil-feature amount Feature1(t_(n1))equals setting the weight w₂ to 0.

If the system control circuit 112 determines that the eyes are in thepredetermined eye state, that is, the eyes are narrow (YES in stepS303), it calculates the eye open/closed degree by using both thepupil-feature amount and feature-change amount by the same method asthat in the first embodiment (step S305). More specifically, the systemcontrol circuit 112 calculates the eye open/closed degreeBlinkDegree(t_(n1)) by using the pupil-feature amount Feature1(t_(n1))calculated from one image at time t=t_(n1)[s] and the feature-changeamount Feature2(t_(n1)). The feature-change amount Feature2(t_(n1))represents the difference between the pupil-feature amountFeature1(t_(n1)) and a pupil-feature amount Feature1(t_(n1-n1′))(1≦n_(1′)<n₁) calculated from a previously input image. In this case,the weights w₁ and w₂ are set to 1:1.

After that, the system control circuit 112 determines the eyeopen/closed state by executing threshold processing for the calculatedeye open/closed degree BlinkDegree(t_(n1)), and sends back thedetermination result (eye closed or eye open) (step S306). Then, theprocess ends.

As described above, according to the second embodiment, if the eyes of aperson, like person A shown in FIG. 13, are not in a predetermined eyestate, that is, are not narrow, the eye open/closed degreeBlinkDegree(t_(n1)) is calculated using only a feature amount obtainedfrom one image. If the eyes of a person, like person B shown in FIG. 13,are in a predetermined eye state, that is, are narrow, the eyeopen/closed degree BlinkDegree(t_(n1)) is calculated using a featureamount obtained from one image and a feature-change amount. By switchingthe eye open/closed degree calculation method on the basis of eye state,the eye open/closed degree can be determined at high precisionregardless of individual differences in eye size. Accordingly,high-precision detection of the eye open/closed degree can be achievedat minimum processing cost.

Third Embodiment

The third embodiment will be explained. The arrangement of an apparatusin the third embodiment is the same as that in FIG. 1 described in thefirst embodiment, a description thereof will be omitted, and onlydifferences from the first embodiment will be explained.

FIGS. 14A and 14B show an example of a processing sequence in an imagesensing apparatus 100 according to the third embodiment. Processes instep S400 to S408 are the same as those in steps S100 to S108 in FIG. 2Ain the first embodiment, and a description thereof will be omitted.

If the user presses the first switch (focusing switch) in step S409 (YESin step S409), face detection, AE/AF control, and through displayprocessing are executed similarly to the first embodiment (steps S410 toS412). After the through display, the pupil-feature amount is calculatedfor a predetermined time (step S413). More specifically, by the samemethod as that in the second embodiment, the pupil-feature amount iscalculated from frame images sequentially input within a predeterminedtime (time t=t₂[s] to time t=t_(n2)[s]).

If the user presses the second switch (shooting switch) (YES in stepS414), shooting is done (step S415). In this case, shooting is performedregardless of the eye open/closed state. At this time, a sensed image istemporarily stored in a memory 108 shown in FIG. 1. In addition to thesensed high-resolution image, the memory 108 also stores alow-resolution image compressed for use in eye open/closeddetermination.

If the user does not press the second switch in step S414 (NO in stepS414), face detection (step S420) and through display (step S421) areexecuted again. Then, the process returns to step S413.

If shooting is done in step S415, a system control circuit 112 performseye open/closed determination using the compressed low-resolution image(step S416). Details of the eye open/closed degree determination will beexplained later.

The system control circuit 112 displays a quick review on the imagesensing apparatus 100 (step S417). At this time, if it is determined inthe eye open/closed degree determination that the eyes are closed, amessage such as “would you like to shoot again?” is displayed to promptthe user to shoot again. It is also possible to force the user to shootwithout any display which prompts him to shoot again.

If the user does not want to shoot again (NO in step S418), the systemcontrol circuit 112 records, in a flash memory or the like, thehigh-resolution sensed image stored in the memory 108 (step S419). Then,the process ends. If the user wants to shoot again (YES in step S418),the system control circuit 112 performs face detection for the nextframe image (step S420), and a through display (step S421). After that,the process returns to step S413.

Details of the eye open/closed degree determination in step S416 shownin FIG. 14B will be explained with reference to FIG. 15. In the thirdembodiment, to prevent a repetitive description, a description ofdetailed processing (e.g., methods of detecting and calculating apupil-feature amount) described in the first and second embodiments willbe omitted, and only a rough sequence representing the difference willbe explained.

When this processing starts, the system control circuit 112 refers topupil-feature amounts Feature1(t₂), . . . , Feature1(t_(n2)) calculatedin step S413 (FIG. 14B) from time t=t₂[s] to time t=t_(n2)[s]. Thesystem control circuit 112 estimates a state in which the eyes of a facedetected in step S410 (FIG. 14A) or step S420 (FIG. 14B) are open (stepS500). Similar to the second embodiment, whether the eyes are in apredetermined eye state is determined by comparing, with a predeterminedthreshold L_(th), the width L between the maximum and minimum values ofthe pupil-feature amount Feature1(t) within a predetermined time. Notethat the pupil-feature amounts Feature1(t₂), . . . , Feature1(t_(n2))from time t=t₂[s] to time t=t_(n2)[s] are stored in the memory 108 inFIG. 1, as described above.

If the system control circuit 112 determines that the eyes are in thepredetermined eye state, that is, are narrow (YES in step S501), itexecutes weight determination (step S502). Details of the weightdetermination will be described later. After the weight determination,the system control circuit 112 performs processing such as weightingsimilarly to the first embodiment, and then calculates a final eyeopen/closed degree BlinkDegree(t_(n2)) (step S503). That is, the systemcontrol circuit 112 calculates the eye open/closed degree using both thepupil-feature amount and feature-change amount.

If the eyes are not in the predetermined eye state, that is, are notnarrow (NO in step S501), the system control circuit 112 performsprocessing such as weighting similarly to the first embodiment, and thencalculates the final eye open/closed degree BlinkDegree(t_(n2)) (stepS503). Also in this case, the system control circuit 112 calculates theeye open/closed degree using both the pupil-feature amount andfeature-change amount.

Thereafter, the system control circuit 112 determines the eyeopen/closed state by executing threshold processing for the calculatedeye open/closed degree BlinkDegree(t_(n2)), and sends back thedetermination result (eye closed or eye open) (step S504). Then, theprocess ends.

Details of the weight determination in step S502 shown in FIG. 15 willbe explained with reference to FIG. 16. In the following description,thresholds which satisfy a relation “L_(th1)>L_(th2)” are used todetermine weights. Note that the weights are determined using a tableprepared in advance.

When this processing starts, the system control circuit 112 determineswhether the width L of the pupil-feature amount Feature1(t) calculatedwithin a predetermined time in step S413 (FIG. 14B) satisfies L>L_(th1).If L>L_(th1) (YES in step S600), the weights w₁ and w₂ are determined tobe 1:1 (step S601), and then the process ends.

If L<L_(th1) (NO in step S600), the system control circuit 112determines whether L_(th1)≧L>L_(th2) holds. If L_(th1)>L>L_(th2) (YES instep S602), the weights w₁ and w₂ are determined to be 1:2 (step S603),and then the process ends. If L<L_(th2) (NO in step S602), the weightsw₁ and w₂ are determined to be 1:5 (step S604), and then the processends. That is, as the width L between the maximum and minimum values ofthe pupil-feature amount Feature1(t) within a predetermined time becomessmaller, the weight w₂ of the eye open/closed degreeBlinkDegree2(t_(n2)) is set larger.

The processes described with reference to FIGS. 15 and 16 change the setweights only when the eyes correspond to a predetermined eye state(narrow eyes), but the present invention is not limited to this. Forexample, when the eyes of a person are not in a predetermined eye state,for example, are large, the weight w₁ may also be set larger than theweight w₂.

FIGS. 17 and 18 are views for explaining effects obtained by setting theweight w₂ of the eye open/closed degree BlinkDegree2(t_(n2)) calculatedbased on the feature-change amount Feature2(t_(n2)) upon detecting apredetermined eye state.

For person A whose eyes are not in a predetermined eye state, that is,are not narrow, as shown in FIG. 17, the weight w₁ ofBlinkDegree1(t_(n2)) and the weight w₂ of the eye open/closed degreeBlinkDegree2(t_(n2)) are set to 1:1.

In contrast, for a person whose eyes are in a predetermined eye state,that is, are narrow, like person B shown in FIG. 17, the weight w₁ ofBlinkDegree1(t_(n2)) and the weight w₂ of the eye open/closed degreeBlinkDegree2(t_(n2)) are set to 1:2. That is, the weight w₂ is setlarger. With this setting, state 2 of person A and state 4′ of person Bcan be distinguished from each other. Further, state 3 of person A whoseeyes are not narrow and state 5 of person B whose eyes are narrow can bedetermined to have the same eye open/closed degree.

For a person whose eyes are very narrow, like person B shown in FIG. 18,the weight w₂ of the eye open/closed degree BlinkDegree2(t_(n2))calculated based on the feature-change amount Feature2(t_(n2)) is setmuch larger. With this setting, the eye open/closed degree can bedetermined regardless of individual differences in eye size.

As described above, according to the third embodiment, it is determinedfrom a change of the pupil-feature amount within a predetermined timewhether the eye size of a person to undergo determination of the eyeopen/closed state is in a predetermined eye state (narrow eyes). Weightsare changed based on the determination result, and then the eyeopen/closed degree is determined. As a result, an eye state can bedetected at higher precision regardless of eye size.

Typical embodiments of the present invention have been described above.However, the present invention is not limited to the above-describedembodiments and embodiments shown in the accompanying drawings, and canbe properly modified without departing from the scope of the appendedclaims.

For example, in the second and third embodiments, a state in which theeyes are narrow is determined as a predetermined eye state. However, thepresent invention is not limited to this, and a state in which the eyesare large may also be determined as a predetermined eye state. Also inthis case, the same processing as that described above can be adopted.

In the second and third embodiments, a predetermined eye state isestimated by comparing, with the predetermined threshold L_(th), thewidth L between the maximum and minimum values of the pupil-featureamount Feature1(t) within a predetermined time. However, the presentinvention is not limited to this. For example, a predetermined eye statemay also be estimated by comparing, with a predetermined threshold, theaverage value of the pupil-feature amount Feature1(t) within apredetermined time.

Some or all processes described in the first to third embodiments mayalso be combined. For example, the processing described in the secondembodiment to switch the method of determining the eye open/closeddegree, and the weight determination described in the third embodimentmay also be executed in combination.

The present invention also includes a case where the functions of theabove-described embodiments are achieved by supplying a software programto a system or apparatus directly or from a remote place, and readingout and executing the supplied program codes by a computer incorporatedin the system or apparatus. In this case, the supplied program is acomputer program corresponding to the flowcharts shown in the drawingsin the embodiments.

Hence, the program codes installed in the computer to implementfunctional processing of the present invention by the computer alsoimplement the present invention. That is, the present invention alsoincludes the computer program for implementing functional processing ofthe present invention. In this case, the program may take the form of anobject code, a program executed by an interpreter, or script datasupplied to an OS (Operating System) as long as the functions of theprogram can be provided.

Various types of the computer-readable storage media may be used forsupplying the computer program.

The program can also be supplied by downloading the computer program ofthe present invention from an Internet homepage to a storage medium suchas a hard disk. The program can also be supplied by grouping programcodes which form the program of the present invention into a pluralityof files, and downloading the files from different homepages.

Further, the functions of the above-described embodiments are alsoimplemented in cooperation with the OS or the like running on thecomputer on the basis of the instructions of the program.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2008-051121 filed on Feb. 29, 2008, which is hereby incorporated byreference herein in its entirety.

1. An information processing apparatus comprising: an input unitconfigured to input an image; a face detection unit configured to detecta face of a person from the image input from said input unit; a firstcalculation unit configured to calculate a feature amount associatedwith an eye open/closed state from the face detected by said facedetection unit; a second calculation unit configured to calculate, as afeature-change amount, a difference between the feature amountcalculated by said first calculation unit and a feature amountcalculated from a face detected from an image input before apredetermined time; and a third calculation unit configured to calculatean eye open/closed degree of eyes of the face detected by said facedetection unit on the basis of the feature amount and the feature-changeamount.
 2. The apparatus according to claim 1, further comprising aweighting unit configured to set weights for the feature amountcalculated by said first calculation unit and the feature-change amountcalculated by said second calculation unit, wherein said thirdcalculation unit calculates the eye open/closed degree of eyes of theface detected by said face detection unit on the basis of the featureamount and the feature-change amount for which said weighting unit setsthe weights.
 3. The apparatus according to claim 2, wherein said firstcalculation unit calculates a feature amount associated with an eyeopen/closed state of a person from faces of the person that are detectedfrom a plurality of images input from said input unit within apredetermined time, and said third calculation unit calculates the eyeopen/closed degree of eyes of the face detected by said face detectionunit on the basis of only the weighted feature amount when a differencebetween a maximum value and a minimum value in a plurality of featureamounts calculated by said first calculation unit from the plurality ofimages has a predetermined relation with a preset threshold.
 4. Theapparatus according to claim 2, wherein said first calculation unitcalculates a feature amount associated with an eye open/closed state ofa person from faces of the person that are detected from a plurality ofimages input from said input unit within a predetermined time, and saidweighting unit changes the weights set for the feature amount and thefeature-change amount on the basis of a plurality of feature amountscalculated by said first calculation unit from the plurality of images.5. The apparatus according to claim 4, wherein said first calculationunit calculates a pupil-feature amount as the feature amount associatedwith the eye open/closed state, and said weighting unit sets the weightfor the feature-change amount to be larger than the weight set for thefeature amount when a difference between a maximum value and a minimumvalue in a plurality of feature amounts calculated by said firstcalculation unit from the plurality of images is smaller than apredetermined threshold.
 6. The apparatus according to claim 4, whereinsaid first calculation unit calculates a pupil-feature amount as thefeature amount associated with the eye open/closed state, and saidweighting unit sets the weight for the feature amount to be larger thanthe weight set for the feature-change amount when a difference between amaximum value and a minimum value in a plurality of feature amountscalculated by said first calculation unit from the plurality of imagesis larger than a predetermined threshold.
 7. The apparatus according toclaim 1, further comprising a determination unit configured to performeye open/closed determination by performing threshold processing for theeye open/closed degree calculated by said third calculation unit.
 8. Aneye open/closed degree determination method comprising: inputting animage; detecting a face of a person from the image input in theinputting the image; calculating a feature amount associated with an eyeopen/closed state from the face detected in the detecting the face;calculating, as a feature-change amount, a difference between thefeature amount calculated in the calculating the feature amount and afeature amount calculated from a face detected from an image inputbefore a predetermined time; and calculating an eye opening/closingdegree of eyes of the face detected in the detecting the face on thebasis of the feature amount and the feature-change amount.
 9. Acomputer-readable storage medium storing a computer program, the programcausing a computer to function as an input unit configured to input animage, a face detection unit configured to detect a face of a personfrom the image input from said input unit, a first calculation unitconfigured to calculate a feature amount associated with an eyeopen/closed state from the face detected by said face detection unit, asecond calculation unit configured to calculate, as a feature-changeamount, a difference between the feature amount calculated by said firstcalculation unit and a feature amount calculated from a face detectedfrom an image input before a predetermined time, and a third calculationunit configured to calculate an eye open/closed degree of eyes of theface detected by said face detection unit on the basis of the featureamount and the feature-change amount.
 10. An image sensing apparatuscomprising: an input unit configured to input an image; a face detectionunit configured to detect a face of a person from the image input fromsaid input unit; a first calculation unit configured to calculate afeature amount associated with an eye open/closed state from the facedetected by said face detection unit; a second calculation unitconfigured to calculate, as a feature-change amount, a differencebetween the feature amount calculated by said first calculation unit anda feature amount calculated from a face detected from an image inputbefore a predetermined time; a third calculation unit configured tocalculate an eye open/closed degree of eyes of the face detected by saidface detection unit on the basis of the feature amount and thefeature-change amount; a determination unit configured to perform eyeopen/closed determination by performing threshold processing for the eyeopen/closed degree calculated by said third calculation unit; and animage sensing unit configured to shoot on the basis of a result of eyeopen/closed degree determination by said determination unit.